{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Neural networks
How they work and their future in science\n", "\n", "Tom Charnock\n", "\n", "Institut d'Astrophysique de Paris\n", "


\n", "\n", "
\n", "
\n", " \"Sorbonne\n", "
\n", "
\n", " \"ANR\"\n", "
\n", "
\n", " \"IAP\"\n", "
\n", "
\n", " \"CNRS\"\n", "
\n", "
\n", " \"Aquila\"\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## What is a neural network?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Neural networks are arbitrary fittable non-linear functions\n", "
\n", "\n", "
\n", "
$$\\mathscr{f} : {\\bf d}\\to\\boldsymbol{\\tau}$$
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### There are many different architectures\n", "
\n", "
\n", " Convolutional neural networks\n", "

\n", " \n", "
\n", "
\n", " Recurrent neural networks\n", "

\n", " \n", "
\n", "
\n", "For any given dataset there is generally an obvious choice of architecture" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## How to fit a neural network" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "

\n", "
Approximate a function, $\\mathscr{g} : {\\bf d}\\to\\boldsymbol{\\theta}$,
with a neural network, $\\mathscr{f} : {\\bf d}\\to\\boldsymbol{\\tau}$
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Probabilistic interpretation of a neural network\n", "
\n", "\n", "How likely is any network going to provide the expected function output?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "This surface is defined by our choice in distance measure which is called the cost or loss function" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Obtaining informative outputs\n", "$$\\mathcal{P}(\\boldsymbol{\\theta}|{\\bf d})= \\int d\\boldsymbol{\\omega}d\\boldsymbol{\\alpha}~\\mathcal{L}(\\boldsymbol{\\theta}|{\\bf d},\\boldsymbol{\\omega},\\boldsymbol{\\alpha})\\mathcal{P}(\\boldsymbol{\\omega},\\boldsymbol{\\alpha})$$\n", "

\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Maximum likelihood estimate of weights\n", "

\n", "\n", "

\n", "$$\\boldsymbol{\\omega}^\\textrm{MLE}=\\underset{\\boldsymbol{\\omega}}{\\textrm{argmax}}\\left[\\mathcal{L}(\\boldsymbol{\\{}\\boldsymbol{\\theta}\\boldsymbol{\\}}^\\textrm{train}|\\boldsymbol{\\{}{\\bf d}\\boldsymbol{\\}}^\\textrm{train}, \\boldsymbol{\\omega}, \\boldsymbol{\\alpha}^*)\\right]$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Prediction using neural networks\n", "

\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Why do we use neural networks?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### They're quick and (relatively) cheap" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### Like how FFT are more efficient than directly evaluating Fourier Transforms\n", "#### Neural networks can fit extremely complex functions to run blazingly quickly" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Super-resolution and super-approximation of dark matter simulations\n", "
\n", "
\n", "
Emulating high resolution N-body simulations
\n", " \n", "
\n", "
\n", "  \n", "
\n", "
\n", "
Inpainting high quality observable quantities
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Neural networks are driven by data\n", "#### i.e. they can be fit with data where there is no description via a physical model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "
\n", "
Generating images of faces
\n", "
\n", " \n", "
\n", "
\n", "  \n", "
\n", "
\n", "
Detecting tumours for aided cancer discovery
\n", "
\n", " \n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "Maybe put galaxy classification test?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Physical models from data models" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Data models are the description that include all effects that generate the data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "

Initial conditions

\n", "

Physical model

\n", "
\n", "
\n", "

Instrumental effects

\n", "

Other stochasticity

\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### Neural networks are great at learning the data model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### We can build neural networks to deal with parts of the data model we don't understand\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Extracting physical models from the data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Correlations between interesting observables and the processes that create them" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Can we build neural networks that are informative about these processes?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Neural physical engines" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Building symmetries into neural networks" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", " \n", "
\n", "

\n", "
\n", "

\n", "
\n", "
\n", " \n", "
\n", "

\n", "
\n", " \n", "

" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Decompose neural networks into well structured, orthogonal, algorithms\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Find what part of the data is informative about a given result\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", " Use visualising attention to perturb the network to find out what is most sensitive\n", "
\n", "
\n", " Because of the orthogonal nature of the neural physical engine we can relate physical structures in the data to outcomes from the trained network\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Statistical understanding of neural networks" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "\n", "
There is no sense in trust in how correct any output of the network is to be correct
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Choosing a neural network with weights at their maximum likelihood estimates degenerates the posterior\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Local maximum likelihood estimates\n", "
\n", "\n", "Different results are obtained with each network, none with a sense of trust" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# No interpretability and no trust = no science" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Bayesian neural networks and variational inference\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Posterior predictive distribution of outputs\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Everything is dependent on the training data!\n", "\n", "
\n", "\n", "Classical network : $\\tiny \\mathcal{P}(\\boldsymbol{\\omega},\\boldsymbol{\\alpha}|\\boldsymbol{\\{}{\\bf d},\\boldsymbol{\\theta}\\boldsymbol{\\}}^\\textrm{train}) \\to \\delta(\\boldsymbol{\\omega}-\\boldsymbol{\\omega}^\\textrm{MLE},\\boldsymbol{\\alpha}-\\boldsymbol{\\alpha}^*)$\n", "\n", "Variational inference : $\\tiny \\mathcal{P}(\\boldsymbol{\\omega},\\boldsymbol{\\alpha}|\\boldsymbol{\\{}{\\bf d},\\boldsymbol{\\theta}\\boldsymbol{\\}}^\\textrm{train}) = \\mathcal{Q}(\\boldsymbol{\\omega}|\\boldsymbol{\\nu}^\\textrm{MLE}, \\boldsymbol{\\alpha}^*, \\boldsymbol{\\{}{\\bf d},\\boldsymbol{\\theta}\\boldsymbol{\\}}^\\textrm{train})$\n", "\n", "Bayesian networks : $\\tiny \\mathcal{P}(\\boldsymbol{\\omega},\\boldsymbol{\\alpha}|\\boldsymbol{\\{}{\\bf d},\\boldsymbol{\\theta}\\boldsymbol{\\}}^\\textrm{train}) =\\prod_i^{n_\\textrm{train}}\\mathcal{L}(\\boldsymbol{\\theta}^\\textrm{train}_i|{\\bf d}^\\textrm{train}_i,\\boldsymbol{\\omega},\\boldsymbol{\\alpha})p(\\boldsymbol{\\omega},\\boldsymbol{\\alpha})$\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## If the training data, network optimisation, or (pretty much) anything else is not perfect, the inference will be biased" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Likelihood-free inference" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### We have a model with which we can simulate the data \n", "- Physical model\n", "- Generative models based on neural networks\n", "\n", "### We cannot (easily) describe the data\n", "- The likelihood of obtaining data is unknown/complex\n", "- There isn't even a well defined model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Use the same neural network to summarise observations and simulations\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Generate simulations drawn from prior and summarise\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Make a kernel density estimate (or use ABC) on the simulations\n", "
\n", "\n", "A slice through at the summarised observation is the approximate posterior" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Posterior distribution of galaxy counts and fluxes in fields

\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Summary" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "##### Neural networks are extremely powerful for science\n", "\n", "##### They speed up computations\n", "\n", "##### They can also allow us to build model-independent models\n", "\n", "##### When used carefully, and in the right setting they give us exact statistically interpretable results\n", "\n", "##### And in the future they will actually inform us on how to build physical models" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Extracting information using neural networks" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Information maximising neural networks\n", "
\n", "
\n", " Which function $\\mathscr{f}: \\mathbb{R}^{\\bf d}\\to \\mathbb{R}^\\boldsymbol{\\theta}$ maximises the Fisher information of the summaries ${\\bf x}$ from that function?\n", "

\n", " $$\\begin{align}\n", " {\\bf F}_{\\alpha\\beta} &= \\left.\\left\\langle\\frac{\\partial^2\\ln\\mathcal{L}({\\bf d}|\\boldsymbol{\\theta})}{\\partial\\theta_\\alpha\\partial\\theta_\\beta}\\right\\rangle\\right|_{\\boldsymbol{\\theta}=\\boldsymbol{\\theta}^*}\\nonumber\\\\\n", " &\\simeq \\left.\\left\\langle\\frac{\\partial^2\\ln\\mathcal{L}({\\bf x}|\\boldsymbol{\\theta}, {\\bf d})}{\\partial\\theta_\\alpha\\partial\\theta_\\beta}\\right\\rangle\\right|_{\\boldsymbol{\\theta}=\\boldsymbol{\\theta}^*}\\nonumber\n", " \\end{align}$$\n", "
\n", "
\n", " And also Gaussianises the data\n", " $$\\mathcal{L}({\\bf x}|\\boldsymbol{\\theta},{\\bf d})\\approx\\mathcal{N}(\\boldsymbol{\\mu}(\\boldsymbol{\\theta}), \\boldsymbol{\\Sigma}(\\boldsymbol{\\theta}))$$\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## We maximise\n", "$$\\ln|{\\bf F}_{\\alpha\\beta}| = \\ln\\left|\\frac{\\partial\\boldsymbol{\\mu}_\\mathscr{f}}{\\partial\\theta_\\alpha}^T{\\bf C}^{-1}_\\mathscr{f}\\frac{\\partial\\boldsymbol{\\mu}_\\mathscr{f}}{\\partial\\theta_\\beta}\\right|.$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Calculate the covariance\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Calculate the derivative of the mean of the summaries with respect to the parameters\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Calculate the Fisher information\n", "\n", "$${\\bf F}_{\\alpha\\beta}=\\frac{\\partial\\mu_\\mathscr{f}}{\\partial\\theta_\\alpha}^T{\\bf C}^{-1}_\\mathscr{f}\\frac{\\partial\\mu_\\mathscr{f}}{\\partial\\theta_\\beta}$$\n", "\n", "This Gaussian form forces the summaries to be Gaussianised" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### And use LFI\n", "
\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "These are the optimal summaries to extract information from the data" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# We have lots of examples of how powerful this tool is throughout scientific domains" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### I'm not going to mention them now because each would take an entire seminar to describe 😉" ] } ], "metadata": { "celltoolbar": "Diaporama", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" }, "rise": { "leap_motion": { "naturalSwipe": true, "pointerColor": "#d80000", "pointerOpacity": 0.5 } } }, "nbformat": 4, "nbformat_minor": 4 }